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Executive summary 


The mission of 360 is to increase the quantity, quality, and diversity of skilled and 
knowledgeable workers in the field of manufacturing. As a recipient of the National 
Science Foundation (NSF) ATE Regional Center award, 360 has engaged in rigorous 
annual evaluation of its work, centered around its primary goals of increasing the 
quantity, quality, and diversity of manufacturing workers; increasing the visibility of 
manufacturing careers; enhancing the educational pipeline for the manufacturing 
industry; and refining the curricula that contributes to the education of manufacturing 
employees. This report summarizes the evaluation outcomes measured during the first 
four years of 360’s implementation, from June 2012 through May 2016. 


During this time frame, evaluation findings suggest that the quantity and diversity 

of workers in 360’s affiliate programs has increased, 360 partners feel the level of 
collaboration among partners is high, manufacturing employers are at least somewhat 
familiar with 360, and that 360 has improved perceptions of the manufacturing industry 
through its youth and community outreach events. 


With respect to challenges, partner ratings of 360’s structural, organizational, and 
operational processes have declined slightly over time; the most frequent areas of 
concern were funding, workload, and timely progress. This first stage of the evaluation 
struggled to measure some aspects of 360’s work, largely due to low survey response 
rates and logistical difficulties. During the next four years, Wilder and 360 should 
identify new strategies to survey students enrolled in 360 programs, measure 360’s 
impact on the quality of workers, and learn whether 360’s outreach activities—such as 
its summer camps and VEX program—directly result in manufacturing-related college 
course enrollment. 
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Introduction 


Over the past four years, through a grant from the National Science Foundation (NSF), 
the 360 Manufacturing and Applied Engineering ATE Regional Center of Excellence 
(360) has been working collaboratively to recruit, educate, and train workers for careers 
in manufacturing. 360 works with partner institutions in the Minnesota State colleges and 
universities (Minnesota State, formerly MnSCU) system to prepare students for careers. 
360 also works with industry leaders to attract and develop the highly skilled workers 
needed to keep Minnesota businesses competitive and profitable. This work has been 
especially important recent years, as the grant came on the heels of the Great Recession. 
In addition, the manufacturing industry, both pre- and post-recession, has struggled to fill 
positions with qualified workers. 


The overall mission of 360 is to increase the quantity, quality, and diversity of skilled 
and knowledgeable workers in the field of manufacturing. To accomplish this mission, 360 
developed several program objectives to be implemented over the course of the grant; those 
objectives are: 


= Increased visibility: Improve the image of manufacturing careers and the 
manufacturing sector. 


= Enhanced pipeline: Prepare students for rigorous manufacturing technician-level 
programs and entice them to pursue manufacturing careers, and implement strategies 
to target under-represented student groups. 


= Industry-driven curricula: Refine curricula that are relevant to today and 
tomorrow’s manufacturing industry. 


Over the course of the four-year grant (June 2012 through May 2016), 360 has completed 
many activities to help meet these objectives and ultimately increase the quantity, quality, 
and diversity of workers in the field of manufacturing. In order to measure the impact of 
its work, 360 asked Wilder Research to conduct a rigorous and unbiased evaluation of the 
project. Evaluation data were shared with multiple stakeholders, including 360 leadership, 
NSF, and others in the field of advanced technical education to help provide timely and 
useful feedback. 


The following report details the cumulative evaluation findings over the past four years 
and is organized around the logic model that was developed at the beginning of the 
project. The logic model and additional information, such as the research methodology, 
can be found in the Appendix. 
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Overall impact of 360 


The ultimate goal of 360 is to increase the quantity, quality, and diversity of skilled 
and knowledgeable workers in the field of manufacturing. By increasing the visibility of 
manufacturing, as well as enhancing the curricula and pipeline for technician-level programs, 
it appears that 360 has met at least part of this goal. Over the past four years, there has 
been an increase in the quantity and diversity of graduates from 360-related programs, but 
evidence related to the improvement in graduate quality is less certain. 


Quantity of workers 


In Wilder’s 2016 survey of regional manufacturing employers, 44 percent of respondents 
believed that 360 had at least some impact on the quantity of graduates from 360-affiliated 
manufacturing and engineering programs (Figure Al). Furthermore, graduate data from 
the 360 Core schools (the 10 original partner institutions) shows a 33 percent increase in 
graduates between the 2010-12 baseline and 2015 (Figure 1). This is an increase to 606 
graduates in 2015 compared to an average of 457 during 2010-12. During the same period, 
the total number of system-wide graduates in the same 360-related programs (but not at 360 
Core institutions) increased 20 percent, from an average of 1,094 per year during 2010-12 
to 1,314 in 2015.This increase suggests that 360’s strategy may have a positive impact on 
the number of graduates. 


The increase between baseline and 2015 was driven by an increase in Precision Production 
and Diploma/Associate graduates. For 360 Core institutions, the number of 360 Precision 
Production graduates increased by 60 percent, compared to only 39 percent among other 
schools. 360 Core schools also experienced a 40 percent increase in Diploma/Associate 
graduates, compared to an 11 percent increase among other schools. 


Tracking enrollment of student majors helps to provide an early indication of successful 
inspiration and recruitment, and a healthy pipeline of students who will potentially become 
the graduates in the future. The 360-related programs experienced an increased number 
of majors during 2014, up 40 percent to 1,554 from an average of 1,112 during 2010-12 
(Figure 1). 
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1. 360-related program enrollments and graduates by year 


2010-12 % increase, 2015 
Baseline 2013 2014 2015 over baseline 
360 graduates 457 525 637 =606 33% 
Total graduates in 360 programs 
at non-partner institutions 1,094 1,066 1,208 1,314 20% 
360 majors 1,112 1,335 1,554 NA 40%* 


Note. Asterisk denotes comparison of 2014 to baseline average. All graduate data reflects the ten “Core” 360 partner 
institutions; major data reflect all ten except Bemidji State. Data on the number of graduates come from Minnesota State’s 
ISRS data system; enrollment information comes from 360. More information available in the Appendix. 


Quality of workers 


In terms of quality, it is difficult to say definitively if 360-affilliated workers are of a 
higher quality than other workers in manufacturing fields. Currently, the best source of 
data regarding 360-impacted worker quality is the employer survey conducted earlier this 
year. About half of respondents believed 360 had at least some impact on the quality of 
workers (10% said “a big impact,” 44% “some impact”; Figure A1). 


Over three-quarters (77%) of respondents had hired manufacturing or engineering graduates 
from a 360-affiliated school in the past year. When asked to compare those graduates 
with other new employees on a series of 14 types of skills and qualities that are important 
for manufacturing workers, majorities of respondents rated 360 graduates’ skills as equal 
to or higher than other new employees (Figure 2). 360’s impact was highest on technical 
skills and general preparedness, with roughly three in ten respondents reporting that 
graduates from 360-affiliated schools had better technical skills related to the job (34%) 
and were generally better prepared for the job (28%). Very few respondents reported that 
non-360 graduates were more skilled than 360 graduates in any area. 
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2. Comparing the skills of graduates, 2016 (N=35-40) 


How would you compare the skills of 


360 graduates 


Other 


recent graduates from 360-affiliated are more They have graduates are 
colleges to other new employees your _ skilledinthis | equal skills in more skilled Don’t 
company has hired in the past year? area this area in this area know 
Technical skill related to the job 34% 40% 8% 18% 
General preparedness for the job 28% 45% 10% 18% 
A positive attitude 26% 49% 0% 26% 
Problem solving 22% 49% 5% 24% 
Working effectively with a team 20% 51% 0% 29% 
Completing work in accordance with 
quality standards 19% 51% 3% 27% 
Correct use and operation of equipment, 
tools, and materials 18% 53% 5% 24% 
Showing respect for other employees 17% 61% 0% 22% 
The ability to accept advice and 
constructive criticism 17% 53% 6% 25% 
Creativity and innovation 14% 54% 5% 27% 
Continuously improving work 
_ performance 11% 64% 0% 25% 
Strong written communication 8% 53% 3% 37% 
_Completing tasks on time 8% 10% 0% 22% 
Strong verbal communication 5% 55% 5% 34% 


Diversity of workers 


In 2015, one-third (32%) of employers surveyed felt that 360 had “some” or “a big” 
impact on the diversity of graduates from 360-affiliated manufacturing and engineering 


programs (Figure A1). 


Administrative data show some positive diversity outcomes when 360 Core schools are 


compared to others in the Minnesota State system. 360 tracks graduate diversity on four 
dimensions: race, gender, first generation college student status, and graduate age (under 


age 35 vs. age 35 and older). In three of the four measures (all but graduate age), the 
increase in diverse graduates has outpaced the increase in other graduates within the 360 


Core schools (Figure 3). Most remarkably, compared to baseline, the number of graduates 
of color more than doubled, from an average of 57 during 2010-12 to 117 in 2015; a 105 
percent increase in graduates of color, compared to a 24 percent increase in white graduates. 
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3. 360 Core partner graduate diversity by year 


2010-2012 % increase 
Baseline 2013 2014 2015 over baseline 
White graduates 388 428 521 480 24% 
Graduates of color 57 85 102 117 105% 
Male graduates 411 464 601 557 11% 
Female graduates 44 37 35 | 49 | 36% 
Not first-generation graduates 319 375 = 4540— 20 32% 
First generation graduates 105 124 163 164 56% 
Graduates under age 35 335 — 397 #476 £459 | 37% 
Graduates age 35 and older 121 126 158 147 21% 


Note. Percent increase over baseline compares the 2015 number to the 2010-2012 baseline average. Due to missing data, 
adding the number of graduates in this list (e.g., under 35 and 35 and older) may not sum to the total provided in tables 
showing the total number of graduates. 


Between the baseline years and 2015, 360 Core programs outpaced other Minnesota State 
programs on all four diversity measures (Figure 4). Again, 360 experienced its largest 
increases in graduates of color (105% increase vs. 39% increase in non-360 Core programs). 
Of note, 360 Core programs experienced growth in the number of first generation college 
student graduates (56%) and graduates age 35 and older (21%), whereas other Minnesota 
State programs saw declines in both populations (7% decrease for both). 


4. Percent increase over baseline, 2010-2012 to 2015 


360 Core Other 

programs programs 
White graduates 24% 15% 
Graduates of color 105% 39% 
Male graduates 11% 20% 
Female graduates 36% 19% 
Not first-generation graduates 32% 31% 
First generation graduates 56% -7% 
Graduates under age 35 37% 30% 
Graduates age 35 and older 21% -7% 


Note. Percent increase over baseline compares the 2015 number to the 2010-2012 baseline average. “Non-360 Core graduates 
system-wide’ refers to graduates of Minnesota State schools that were not affiliated with 360 at the time of the grant. 
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Perceptions of stakeholders about 360 


Wilder also looked at the impact of 360 on some of its key stakeholders, including project 
partners and employers. 


Partnership 


As part of Wilder’s annual partner survey, respondents were asked a series of questions 
adapted from the Collaboration Factors Inventory!. These questions are intended to gauge 
the overall strength of the collaboration activities and structure from the perspective of 
those involved. Overall, respondents gave very favorable ratings to 360’s collaboration 
and operations. A score of five is “perfect,” higher than four indicates a “strength,” a 
score of between three and four is “borderline,” and a score of less than three indicates a 
“weakness.” More information about how these scores are calculated and the questions 
that contribute to each are available in the Appendix. 


Operations measure 


Figure 5 shows the aggregated responses to the questions that focus on structural, 
organizational, and operational aspects of the 360 partnerships. These questions target 
factors that indicate successful operations, specifically related to having the right 
leadership strategies, adequate funding, and making appropriate progress toward its 
goals. The average operations score increased between 2013 and 2014, then decreased 
between 2014 and 2016 from 4.30 to 3.83. The most recent survey year, 2016, was the 
first time that the mean score dipped into the borderline range; however, 2016 results 
should be interpreted with caution due to the low response rate and small number of 
respondents (N=10). 


'  Mattessich, P., Murray-Close, M., & Monsey, B. (2001). Wilder Collaboration Factors Inventory. St. Paul, 
MN: Wilder Research. 
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5. Operations measure, Years 1-4 


Perfect 


4.30 


4.09 3.83 
Strength 


Borderline 


2013 2014 2015 2016 


Operations score 


Across all four years of the survey, the two highest ranked operations questions have 
been “360 is doing work that would not otherwise be happening” (average score: 4.42) 
and “360 has the right leadership to succeed” (average score: 4.36). There was a slight 
decline in both measures during 2016; however, this is unsurprising, given the small 
number of survey responses (N=10) and 360’s recent leadership transition, in which a 
new executive director was hired a few weeks prior to the survey. 


Across the four survey years, the three lowest-ranked operations questions have been 
“360 has taken on the right amount of work” (average score: 3.82), “360 has adequate 
funding to complete the work” (average score: 3.86), and “360 is progressing at the right 
pace” (average score: 3.99)—the only three questions with average scores under 4.00. 
Qualitative responses further illuminate some of these points. For example, partners 
suggest that 360 may be taking on too much work. In the 2015 survey, one partner said, 
“there are too many balls in the air, and not enough time to step back and manage effectively 
what is currently happening;” another suggested that 360 has struggled with remaining 
“focused on a few main goals, not spreading [themselves] too thin.” With respect to 
funding, partners suggested that there was not adequate funding for the work, some 
specifically speaking to the relatively low salary level of staff. 


Collaboration measure 


Figure 6 shows the average score across the six collaboration questions for each of the 
four years of the partner survey. The collaboration measure consistently remained above 
score of 4.00 (“strength”) across the four years of the evaluation. The pattern for average 
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collaboration score mirrors the pattern for operations score, though the average 
collaboration score increased slightly in 2016. Again, the 2016 results should be 
interpreted with caution due to the low response rate and small number of respondents 


(N=10). 
6. Collaboration measure, Years 1-4 


Perfect 
4.22 


4.04 


Strength 


Borderline 


2013 2014 2015 2016 


Collaboration score 


Over time, the strongest responses related to collaboration have typically been “discussions 
among 360 partners encourage sharing of diverse points of view” (average score: 4.28) 
and “T have a clear sense of what 360 is trying to accomplish” (average score: 4.23). 


The only item in the collaboration set with an average score of less than 4.00 is “360 
partners have a clear sense of their roles and responsibilities”; individual year ratings for 
this item never reached the “strength” level (average of 4.00 or greater), with averages 
ranging from a low of 3.75 (in 2015) to 3.90 in 2016. 
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Employers 


In a survey of employers conducted in early 2016, respondents were asked about their 
familiarity with 360, as well as its affiliated programs and partner organizations. The 
majority of respondents said they were at least somewhat familiar with 360 (65%); 
although over one-third were not very or not at all familiar (Figure 7). 


7. Familiarity with 360 and affiliated programs and partners, 2016 (N=52) 


How familiar are you with the Very Somewhat Not very Not at all 
following entities? familiar familiar familiar familiar 
The Statewide Tour of Manufacturing 40% 19% 17% 23% 


Manufacturing and engineering 
programs at Minnesota State 


Colleges and Universities (MnSCU) 37% 46% 10% 8% 
Dream It. Do It. 27% 15% 19% 39% 
The 360 Manufacturing Center of 

Excellence 15% 50% 25% 10% 
The VEX Robotics program 15% 23% 27% 35% 
360 eTECH Online and Hands-on 

Manufacturing Education 6% 40% 23% 31% 
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Improved perceptions of 
manufacturing 


Over the past four years, 360 has held a number of events to improve perceptions and 
increase the visibility of manufacturing careers among both children and adults. Most 
notable is the Minnesota Statewide Tour of Manufacturing. Every fall, 360 and its partners 
coordinate with manufacturing businesses across Minnesota to provide tours of their 
facilities for students, educators, job seekers, other manufacturers, and the general public. 
Through these tours, as well as its summer camps and the VEX Robotics competition, 360 
hopes to improve awareness, perceptions, and interest in manufacturing and Science, 
Technology, Engineering, and Math (STEM). 


After each event of these events, participants are asked to complete a survey to gauge the 
event’s impact on participants’ awareness, perceptions, and interest in manufacturing 
careers, as well as interest and confidence in STEM. Participants are asked to rate their 
familiarity or opinion of each of these both before and after the program. More information 
on the questions and response options is available in the Appendix. 


Survey results across the four years suggest 360 has had a substantial impact on the visibility 
of manufacturing careers and STEM, particularly on the perceptions of manufacturing 
careers. For the measures selected: 


= 360 has made the greatest impact on perceptions of manufacturing: 64 percent of 
youth and 78 percent of adults surveyed increased their perception of manufacturing. 


= Among youth, 360 has made the least impact on interest in manufacturing careers: 
46 percent of youth surveyed increased their interest in manufacturing careers. 


= Among adults, 360 has made the least impact on interest in STEM: 31 percent of 
youth surveyed increased their interest in manufacturing careers. 


= Youth events had a similar impact on perceptions among boys and girls, except for 
the STEM measures: a greater percentage of girls increased their interest and 
confidence in STEM compared to boys. 


These outcomes are explored in greater detail below. 
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Improvement in perceptions of manufacturing 


Figures 8 and 9 show the improvement in perceptions of manufacturing for each measure 
among youth and adult participants. 


The in percentage reflects participants who were not already highly interested in the 
topic—that is, it omits those who said they were highly engaged in the measure (e.g., 
very aware of manufacturing careers) both before and after the program. 


Figure 8 shows the impact of 360 camps, VEX, and the Tour of Manufacturing among 
youth participants. Among all youth participants surveyed, 360 had the most influence on 
the perception of manufacturing careers (64% increased), followed by awareness of 
manufacturing careers (57% increased) and confidence in STEM (57% increased). 
Impact was somewhat lower with respect to interest in STEM (50% increased) and 
interest in manufacturing careers (46% increased). 


8. Percent of youth participants who reported increased visibility, 2013-2016 


2013 2014 2015 2016 All Total youth 
(N=130- (N=116- (N=141- (N=40- (N=459- who 
179) 211) 230) 72) 692) increased 
Awareness of manufacturing 
careers 56%  — 49% 68% 49% 57% 366 
Perception of manufacturing 
Careers 68% 61% 69% 53% 64% 350 
Interest in manufacturing 
_ careers 50% 36% 55% 39% 46% 319 
Interest in science, technology, 
engineering, and math (STEM) 56% 40% 51% 65% 50% | 236 


Confidence in science, 
technology, engineering, and 
math (STEM) 56% 53% 60% 57% 57% 261 


Notes. The total number of youth who reported an increase is a minimum, based on the number of respondents surveyed. This table 
represents the combined responses of youth participating in the Tour of Manufacturing, VEX, and Summer Camps surveys. Denominator 
for percentage omits those that “remained high’—e.g., those who were interested in STEM “a lot” both before and after the program. 
More information available in the Appendix. 
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Figure 9 shows a similar pattern among adults, though the percentage increases were 
notably lower than for youth. This could be because adult events are less intensive than 
youth events, spend less time on topics that would increase interest in manufacturing 
careers or STEM, or because adults are less likely to change their mind about manufacturing 
careers or STEM. Note that about two-thirds of the adults surveyed (67%) participated in 
a Tour of Manufacturing event, a one-day tour of a manufacturing facility designed by 
manufacturing employers, which could have a limited impact compared to the more 
intensive, 360-directed youth activities (e.g., a multi-day camp). The remaining adults 
supported youth in the VEX Robotics Tournament, either as coaches or parents. 


The greatest impact on adults was in the perception of manufacturing careers (78% 
increased), followed by the impact on the awareness of manufacturing careers (49% 
increased). Compared to youth, adult participants were less likely to demonstrate 
increases in interest in manufacturing careers and STEM (31 and 36%, respectively). 


9. Percent of adult participants who increased visibility, 2013-2016 


2014 All Total adults 
2013 (N=104- 2015 (N=253- who 
(N=11-130) 192) (N=69-109) 314) increased 

Awareness of manufacturing 
careers 13% 47% 50% 49% 144 
Perception of manufacturing 
careers 17% 15% 86% 18% 244 
Interest in manufacturing 
careers 38% 33% 41% 36% 114 
Interest in science, technology, 
engineering, and math (STEM) 18% 31% 34% 31% 79 


Notes. The total number of adults who reported an increase is a minimum, based on the number of respondents surveyed. This table 
represents the combine responses for adults responding to the Tour of Manufacturing and VEX surveys. Denominator for percentage 
omits those that “remained high’—e.g., those who were interested in STEM “a lot” both before and after the program. More information 


available in the Appendix. 
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Youth events had similar impact on boys and girls, except for STEM measures 


Figure 10 shows the percentage of youth participants that increased their awareness, 
perception, interest, or confidence in manufacturing or STEM out of those who were not 
highly engaged prior to camp, broken down by gender. 


There was no substantial gender difference between boys and girls with respect to their 
awareness, perception, or interest in manufacturing careers. Of these three measures, both 
boys and girls most frequently improved their perceptions of manufacturing (65-66%), 
demonstrating slightly lower increases in awareness (57-58%) and interest in manufacturing 
careers (45-50%). 


However, girls were more likely to increase their interest and confidence in STEM than 
boys (57% vs. 47% and 64% vs. 53%, respectively). 


10. Percent of youth participants who increased visibility, by gender, 2013- 


2016 
Boys Girls Total 

Measure (N=289-478) (N=152-201) (N=446-679) 
Awareness of manufacturing careers 57% 58% | 57% 
Perception of manufacturing careers 65% 66% | 65% 
Interest in manufacturing careers 45% 50% | 46% 
Interest in science, technology, 

engineering, and math (STEM) 47% 57% 50% 
Confidence in science, technology, 

engineering, and math (STEM) 53% | 64% 57% 


Note. Denominator for percentage omits those who “remained high’—e.g., those who were interested in STEM “a lot” both 
before and after the program. More information available in the Appendix. 
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Enhanced pipeline 


As described earlier, 360 has increased interest in manufacturing among K-12 participants 
in its events, and the number of graduates in manufacturing-related programs at 360 
partner schools has increased. Ultimately, enhancing the pipeline for manufacturing 
workers is all about the things that happen in between these two touch points. An enhanced 
pipeline ensures that students inspired to pursue manufacturing are able to move easily 
and efficiently from interested in a manufacturing career to graduating with a degree in 
manufacturing. 


Many of the aspects of 360 that are addressed in other sections of this report—such as 
refined curricula and improved perceptions of manufacturing—are integral to this goal. 
Understanding how 360 has impacted the pipeline for manufacturing workers requires us 
to understand how graduates’ cumulative education- and employment-related experiences 
contributed to their degree in manufacturing. 


Ideally, measuring this pipeline would involve tracking participants in 360 events and 
programs over time, from K-12 through employment. However, this type of tracking is 
very time and labor-intensive in practice, likely requiring alignment of multiple data 
systems, written consent from parents and students, and substantial analytical work. In 
lieu of this approach, Wilder attempted to complete a survey of students in 360 courses to 
better understand the experiences that led them to pursue a field related to manufacturing. 


In 2016, 360 staff sent a survey link to a selection of professors, requesting that they 
share the link with students. However, only 32 students completed the survey. This is a 
very low response rate, considering that 360 identified approximately 1,500 majors at its 
core partner schools for the year 2014. Wilder and 360 hope to develop strategies to reach 
a greater number of students in the future. Given the low response rate, results should be 
interpreted with caution. 


Despite the low response rate, there was some variety in respondent demographics and 
educational experiences. The majority of respondents were studying Electronics, 
Robotics, Automation, Mechatronics, and/or Industrial Maintenance (47%), with smaller 
proportions studying Machine Tool Technology (22%), Welding and Fabrication (22%), 
or a different subject area (9%). Most students were nearly finished with their programs 
(59%), while 35 percent of students were about halfway through, and 7 percent had just 
started. Respondents came from six different partner schools: Hennepin County Technical 
College (35%), Pine Technical and Community College (23%), South Central College 
(23%), Central Lakes College (13%); two students (6%) said they came from other 
partner schools. Though all respondents were male and most were white (96%), many 
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different age groups were represented: 38 percent were age 18 to 25, 41 percent were age 
26 to 45, and 21 percent were age 46 or older. 


Very few survey respondents specifically identified participation in 360-affiliated K-12 
programs (e.g., summer camps, VEX, etc.), so an accurate assessment of 360’s K-12 
outreach impact on the pipeline is limited. This is unsurprising, because only eleven 
respondents identified themselves as age 18 to 25—the group that would have been most 
likely to access 360 outreach programs. Two respondents had participated in a robotics 
tournament or competition; of those, one specifically noted participation in VEX. 


Respondents were positive about the impact of manufacturing tours. Most respondents 
(90%) had attended a tour of a manufacturing facility. Though none said they attended 
the Statewide Tour of Manufacturing, 85 percent attended a tour organized by their local 
college, 11 percent participated in a tour sponsored by their high school, and 15 percent 
had participated in a different kind of tour. In general, these tours had a substantial 
impact on respondents: 62 percent said the tours impacted their decision to pursue their 
current program or career “a fair amount” (46%) or “a lot” (15%). Even if this impact 
cannot be directly attributed to 360, it does indicate that the Tour of Manufacturing 
strategy is effective in building a pipeline of students. 


While the positive findings from the youth outreach surveys cannot be directly tied to the 
increased enrollment and graduate numbers, we are hopeful that continued attention to 
youth outreach efforts and increasing the visibility of manufacturing careers, through 
such efforts as the popular Statewide Tour of Manufacturing, will continue to help build 
the pipeline of manufacturing workers. 
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Refined curricula 


In addition to increasing the visibility of manufacturing careers and enhancing the 
pipeline of students to workers, 360 worked with industry leaders to refine manufacturing- 
related curricula. Two examples of this work are the creation of a guide for K-12 teachers 
and 26 Career Success Skills modules. 


Teacher guide survey 


In 2014, 360 sponsored the production and dissemination of a toolkit for teachers, called 
Your Future is Made in Manufacturing. This guide was created to help K-12 teachers 
introduce manufacturing careers to their students. To help understand how teachers used the 
toolkit and how the materials might be improved, Wilder sent a survey link to a list of 46 
educators who had downloaded a toolkit online. In total, 22 individuals accessed the 
survey. However, only five respondents had used the toolkit with students, so survey data 
are not particularly helpful in illustrating the impact of the teacher guide. 


Five educators had used the toolkit with students, reaching an estimated 118 students in 
10, 11", and 12™ grades. Those who had not used it said that they were planning to use 
it in the future, had passed it on to other staff, or had not yet had time for the packet. The 
toolkit seemed to have a positive impact on student and teacher perceptions of 
manufacturing. Four out of five educators said that “some” or “many” of their students 
expressed new interest in manufacturing careers after working with the toolkit. All five 
educators said they were more likely to recommend manufacturing careers to their 
students after using the toolkit. 


The survey was administered again in 2015, but with a very low response rate. A survey 
was emailed to 147 addresses that had requested the guide, now called An Introduction to 
Manufacturing in Minnesota Teacher Guide. Only 18 recipients (12%) responded to the 
first question of the survey. Nearly three-quarters of respondents (73%) said they had 
either already used or planned to use the teacher guide with students or employees. At the 
time of the survey, approximately 100 students or employees had already been taught 
using the Teacher Guide. Those who had not yet used the guide expected to teach 
approximately 1,800 students or employees with the guide during the upcoming year. 


Again, because of the low response rate and small number of teachers using the guide, it 
is difficult to comment on the exact impact of the materials developed. 
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Career Success Skills survey 


In order to provide educators and industry with more resources to help students and 
employees develop their professional skills, 360 created 26 Career Success Skills 
learning modules. These modules were developed with industry and faculty feedback to 
improve the abilities of employees, and include topics such as: verbal communication, 
reliability, having a positive attitude, responsibility and accountability, problem solving, 
continuous learning, and critical thinking. Two types of groups can utilize the modules: 


= Intermediaries — the teachers, faculty, industry professionals, and others who access 
the modules and then use them with students, employees, or others 


= End Users — the students, employees, or others with whom the Intermediaries share 
the modules 


Wilder sent a survey to each group to understand how both groups used the 360 Career 
Success Skills learning modules, how the modules can be improved, and the possible 
impact the modules are having on students and employees. It should be noted that the 
response rate for both surveys was quite low (see Appendix for detailed methodology). 


Intermediaries 


The majority of respondents (81%) felt that the 360 Career Success Skills learning modules 
were easy to access online, and the same percentage (81%) had already recommended the 
modules to their colleagues. Over half (57%) of respondents had used the modules with 
End Users, who were most often students (67%). Respondents were asked about the 
effectiveness of the modules in helping End Users improve their skills in specific areas. 
Respondents were most likely to say that the modules were at least somewhat effective in 
helping End Users with the following skills: problem solving (91% very or somewhat 
effective), verbal communication (91%), critical thinking (82%), having a positive attitude 
(82%), and responsibility (82%). 


Several respondents had suggestions for how the modules could be improved, including 
ways to track if students have completed the quizzes or watched the videos; having a 
more polished presentation of the modules, with correct grammar, spelling, and uniformity; 
and more interactive elements, such as assessments and role playing. 


End Users 


Nine of the ten End Users who completed the survey were employed full time, while the 
remaining respondent was unemployed. All respondents felt that the Career Success 
Skills learning modules were effective. Respondents were asked about the effectiveness 
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of the modules in helping End Users improve their skills (the same set of skills listed in 
the Intermediary survey). End Users were most likely to say that the modules were at 
least somewhat effective in helping them improve their skills in the following areas (9 
respondents each): continuous learning, critical thinking, verbal communication, and 
flexibility. When asked which skill they were most interested in improving, the majority 
said verbal communication (78%), followed by continuous learning (67%). 
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Conclusion 


With respect to its primary goals, 360 has demonstrated substantial success, particularly 


with respect to the quantity and diversity of graduates and collaboration among partners. 


Strengths 


The quantity of workers has increased. The number of graduates in 360-affiliated 
majors from 360’s ten original partner institutions as increased 33 percent over the 
baseline average, compared to an increase of 20 percent from other schools. 


The diversity of workers has increased. The number of diverse graduates—diverse 
in terms of race, gender, age, and first generation student status—has grown more at 
360’s ten original partner institutions than at other Minnesota State programs. 


Collaboration among 360 partners remains highly rated. Only one of the six 
collaboration measures received an average rating lower than 4.00, which is the 
threshold for a “strength.” 


Manufacturing employers are familiar with 360. In a survey of 52 regional 
manufacturing employers, nearly two-thirds (65%) were at least somewhat familiar 
with 360. 


360 has improved perceptions of manufacturing in many of its youth and adult 
participants. Perceptions of manufacturing increased in 64 percent of youth and 78 
percent of adults surveyed after participating in one of 360’s youth and community 

outreach programs (the Tour of Manufacturing, VEX, and summer camps). 


For additional consideration 


Partners’ average rating of 360’s structural, organizational, and operational processes 
has declined slightly over time. Three out of eight measures had an average score in 
the “borderline” level across the four years of the survey: “360 has taken on the right 
amount of work,” “360 has adequate funding to complete the work,” and “360 is 
progressing at the right pace.” 


The quality of workers has been difficult to measure. Employers struggle to differentiate 
workers, so it is difficult to determine 360’s impact on graduate quality. 


Surveying students in 360 programs at partner institutions has been challenging, 
primarily due to a variety of logistical concerns. 
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= Surveys designed to assess 360’s goal of refined curricula (the Teacher Guide and 
Career Success Skills surveys) had relatively low response rates, so it is difficult to 
assess the impact of curriculum. However, most of those who had used the tools felt 
that they were at least somewhat effective. 
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VISION: 

A 21° Century education system that prepares individuals to participate 
fully in rewarding careers in manufacturing and which completely meets 
the needs of the region’s manufacturing employers. 


e€ ‘ad Manufacturing and Applied Engineering 


ATE REGIONAL CENTER OF EXCELLENCE 


LOGIC MODEL 


INPUTS ACTIVITIES RESULTS 
ISSUES RESOURCES COLLABORATION IMPLEMENTATION MEASURES IMPACT 
REPORTING / 
ja FEEDBACK STUDENTS AND GRADUATES 
# of graduates -- all manufacturing programs statewide 
# of graduates -- 360° college members manufacturing programs 
e Lack of coordination in systems _EVALUATION/ASSESSMENT # of students in eTECH VISIBILITY 
e Current curricula does not meet EVALUATION Map industry certifications and identify gaps # of graduates in eTECH Improved image of 
industry needs for career ready TEAM i: i employers # of students -- Bemidji Training Institute manufacturing careers 
employees ORGANIZATIONS , rack students/ agen Demographics (age, gender, race/ethnicity, and if first generation) Improved image of 
e University and college easure progress on goals Increased wages/salaries manufacturing sector 
members Balanced Scorecard/Baldrige Satisfaction measures 
e College associates Identify successes and challenges 
e K-12 school system CURRICULUM 
e MnSCU System Office Increased quality of 
° oo en ee IT CHANGING PERCEPTIONS students and graduates 
e Associations an F | thi i 
aaneeraitis COMMITTEE Statewide Dream !t Do !t implementation 


Lack of diverse student 
populations in manufacturing 
programs 
Not enough graduates to meet 
manufacturing industry needs 
PEOPLE 
e Incumbent workers 


e Potential students 
e Public 
e Youth 
e Students 
Manufacturing industry suffers from a 
negative image which is impacting 
the number of individuals considering 
manufacturing careers 
RESOURCES 
e ATE 


e College members 

e Dream !t Do !t (DIDI) 
e Associations and 
foundations 

MnSCU System Office 
e University 

e eTECH 


Developed with the assistance of 
Wilder Research 
June 2013 


= 


=> 


YOUTH 
OUTREACH TEAM 


COMMUNITY 
EAMLE 
OF PRACTICE paces 
PATHWAYS TEAM 


NATIONAL 
VISITING 
COMMITTEE 


TECHNOLOGY 
TEAM 


CURRICULUM 
TEAM 


Regional Dream !t Do !t implementation 
Tour of Manufacturing 
Teacher Toolkit and Adopt-a-School 


YOUTH OUTREACH 
STEM camps and events 
STEM competitions 
Girl-targeted activities 


PATHWAYS 
High school articulation 
Seamless pathways 
Credit for prior learning 
Articulation agreements for veterans 


TOOLS 


TECHNOLOGY 
Web 2.0 
Smart phone and tablet experiments 
Simulation learning 
Social media marketing 
Web-based learning management 


| 


CURRICULUM 
Stackable — Modular coursework 
Customized lab structure 
Workplace readiness/success skills 
Next generation tech and business practices 
Problem-based and inquiry learning 


ENHANCED PIPELINE 


INDUSTRY-DRIVEN CURRICULUM 


CHANGING PERCEPTIONS 

# of participants in Tour of Manufacturing (TOM) 

# of participants more positive towards mfg. careers 

# of businesses hosting tours at the TOM 

# of people joining talent communities 

# of partnerships/people collaborating who haven’t before 
# of interactions between schools and businesses 

# of schools engaged in the Dream !t Do !t (DIDI) program 
# joining social media/talent communities/participation 

# of referrals (web traffic) to DIDI from businesses/schools 
# of blogs/guest posts 

# of times logo is downloaded from DIDI site 

# of times QR code is used 

# of views for online video 

# of website visitors going to college websites 

# of toolkits provided to employers 

# of employers who deliver toolkits to schools and “Adopt- 
a-School” 

# of teachers who receive toolkits 

# of teachers and employer reps trained 

# of students exposed to toolkits 

# of times app is downloaded 


YOUTH OUTREACH 

# of K-12 events and camps/# geared towards girls 

# of attendees at events/# of girls participating 

# of youth more positive towards STEM 

# of youth more confident in STEM 

# of youth more positive towards manufacturing careers 
# of youth more aware of manufacturing careers 

# of adults more positive towards manufacturing careers 


100s MORE 
TECHNICIANS 


GAINFULLY 
EMPLOYED 


PIPELINE 
Increased quantity of 
students and graduates 
Increased diversity of 
students and graduates 


CURRICULUM 

# of students taking modules/non-credit training 

# of hours in non-credit training 

# of students in Career Success Skills curriculum 

# of colleges/programs using Career Success Skills 
# of people attending forums 

# of students in problem-based learning (PBL) 

# of faculty receiving professional development in PBL 
# of college programs implementing PBL 

# of courses impacted by curriculum improvements 
# of documents added to new management system 
# of faculty using new management system 


PATHWAYS 

# of students articulating to BSU 

# of high schools offering eTECH courses 

# of credits received by high school students 

# of high school students taking eTECH courses 
# of workers receiving credit for prior learning 

# of credits received by credit for prior learning 
# of veterans using VETS system/awarded 
credit for service experience/placed in 360 
program 


Quality, quantity, and diversity 


Figure Al summarizes the quantity, quality, and diversity questions from the 2016 
employer survey. 


A1. Overall impact of 360, 2016 (N=50) 


What impact do you think the 360 


Manufacturing Center of Excellence A big Some Little No Don’t 
has had on the following... impact impact impact impact know 
The quantity of graduates from 360- 

affiliated manufacturing and engineering 

programs 2% 42% 12% = 6% 38% 
The quality of graduates from 360- 

affiliated manufacturing and engineering 

programs 10% 44% 2% 4% 40% 
The diversity of graduates from 360- 

affiliated manufacturing and engineering 

programs 2% 30% 14% = 8% 46% 


Source: 2016 Employer Survey 
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Perceptions of manufacturing 


Table A2 outlines the source data for the perceptions of manufacturing data, which 


includes measures related to awareness, perceptions, and interest in manufacturing 


careers. Surveys from 2012 were not included, as consistent questions to measure 


interest, confidence, and awareness had not yet been finalized. 


A2. Youth and adult visibility measures, sources, and years 


Camp 
surveys TOM 
(youth VEX (youth (youth and 

Measure Measure specification only) and adults) adults) 

Youth measures 

Youth awareness of # of youth more aware of 

manufacturing careers manufacturing careers 2013-2015 2013-2016 2013-2015 
# of youth more positive 

Youth perceptions of towards manufacturing 

manufacturing careers careers 2013-2015 2013-2016 2013-2015 
# of youth more 

Youth interest in interested in 

manufacturing careers manufacturing careers 2013-2015 2013-2016 2013-2015 

Youth interest in # of youth more 

STEM interested in STEM 2013-2015 2013-2016 2013-2015 

Youth confidence in # of youth more confident 

STEM in STEM 2013-2015 2013-2016 2013-2015 

Adult measures 

Adult awareness of # of adults more aware of 

manufacturing careers manufacturing careers NA NA 2013-2015 
# of adults more positive 

Adult perceptions of towards manufacturing 

manufacturing careers careers NA — 2013-2016 2013-2015 
# of adults more 

Adult interest in interested in 

manufacturing careers manufacturing careers NA NA — 2013-2015 
# of adults more 

Adult interestin STEM _ interested in STEM NA NA 2013-2015 


The questions and response options for each measure are outlined in Figure A3. 
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A3. Youth and adult visibility questions 


Measure 


Question text 


Valid response options 


Youth/adult awareness of 
manufacturing careers 


Youth/adult perceptions of 
manufacturing careers 


Youth/adult interest in 
manufacturing careers 


Youth/adult interest in 
STEM 


Youth confidence in STEM 
(not measured for adults) 


Think about BEFORE [the event]/ 
NOW, after [the event]. How much 
were you/are you... Aware of 
careers in manufacturing? 


How did you feel about 
manufacturing careers BEFORE 
the camp? How do you feel about 
manufacturing careers NOW? 


Think about BEFORE [the event]/ 
NOW, after [the event]. How much 
were you/are you... Interested in 
manufacturing careers? 


Think about BEFORE [the event]/ 
NOW, after [the event]. How much 
were you/are you... Interested in 
science, technology, engineering, 
or math? 


Think about BEFORE [the event]/ 
NOW, after [the event]. How much 
were you/are you... Confident in 
your ability to do science, 
technology, engineering, or math? 


A lot 
Some 
Very little 


Not at all 
| thought they were good 


| thought they were just ok 
| didn't think they were good 


| didn't think about them 
A lot 
Some 
Very little 
Not at all 
A lot 
Some 
Very little 
Not at all 
A lot 
Some 
Very little 
Not at all 


The following tables outline the complete, year-by-year outcomes for each of these 


measures. The main body of the report examines the percent of respondents who 


increased their interest out of the total number of respondents who did not begin and end 
camp with the highest level of interest, which shows the extent to which 360 was able to 


impact those who were not already highly interested in manufacturing careers and STEM. 


An alternate way to look at success for these surveys is to count those who maintained 
the highest response as an equally positive outcome as increasing interest in less-engaged 


respondents. This alternate measure does not just measure improvement, but also rewards 


360 for attracting and maintaining highly-engaged participants. 


With this approach, all responses can be grouped into positive and neutral/negative 


outcomes. In this section, individuals with positive outcomes either increased or 


maintained a high level of engagement, while those with neutral or negative outcomes 
remained at the same level of engagement (that was lower than the highest level) or 


demonstrated a decrease in engagement. From this perspective, across all 360 events: 
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= 68 percent of youth and 68 percent of adult respondents increased or maintained a 


high level of awareness of manufacturing careers 


= 76 percent of youth and 91 percent of adult respondents improved their perception or 
maintained a high opinion of manufacturing careers 


= 58 percent of youth and 55 percent of adult respondents increased or maintained a 


high level of interest in manufacturing careers 


= 74 percent of youth and 63 percent of adult respondents increased or maintained a 
high level of interest in science, technology, engineering, and math (STEM) 


Figures A4 through A12 show these breakdowns over time. “No change” refers to those 
who maintained a lower level of engagement before and after the event (e.g., some, very 


little, not at all). 


Awareness of manufacturing careers 


A4. Awareness of manufacturing careers among youth, 2013-2016 


2013 2014 2015 2016 All 
(N=230) (N=263) (N=291) (N=100) (N=884) 
Increase in awareness 41% 37% 48% 34% 41% 
Maintained high awareness 26% 24% 30% 31% 27% 
No change in awareness 27% 35% 19% 32% 27% 
Decrease in awareness 6% 4% 3% 3% 4% 


A5. Awareness of manufacturing careers among adults, 2013-2015 


2013 2014 2015 All 
(N=23) (N=262) (N=168) (N=453) 
Increase in awareness 35% 32% 32% 32% 
Maintained high awareness 52% 33% 37% 36% 
No change in awareness 4% 31% 29% 29% 
Decrease in awareness 9% 4% 2% 4% 
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Perceptions of manufacturing careers 


A6. Perceptions of manufacturing careers among youth, 2013-2016 


2013 2014 2015 2016 All 
(N=195) (N=242) (N=262) (N=94) (N=793) 
_ Improvement in perception — 45% 40% 49% 39% 44% 
Maintained positive perception 33% 35% 29% 26% 32% 
_No change in perception 21% | 238% | 21% | 35% 23% 
Worse perception 1% 2% 0% 0% 1% 
A7. Perceptions of manufacturing careers among adults, 2013-2016 
2013 2014 2015 2016 All 
(N=21) (N=313) (N=247) (N=143) (N=724) 
Improvement in perception 33% 32% 32% 41% 34% 
Maintained positive perception 48% 58% 58% 52% 57% 
No change in perception 14% 8% | 11% | T% 9% 
Worse perception 5% 1% 0% 0% 1% 
Interest in manufacturing careers 
A8. Interest in manufacturing careers among youth, 2013-2016 
2013 2014 2015 2016 All 
N=232 N=264 N=292 N=100 N=888 
Increase in interest 38% | 29% 43% 28% 36% 
_ Maintained high interest 23% | 20% 21% | 28% 22% 
No change in interest 32% | 48% | 33% | 41% 38% 
Decrease in interest 1% 3% | 3% 3% 4% 


A9. Interest in manufacturing careers among adults, 2013-2015 


2013 2014 2015 All 

(N=23) (N=257) (N=164) (N=444) 
Increase in interest 22% 25% 27% 26% 
Maintained high interest 43% 25% | 34% | 29% 
No change in interest 26% 46% 37% | 41% 
Decrease in interest 9% 4% 2% 4% 
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Interest in science, technology, engineering, and math (STEM) 


A10. Interest in STEM among youth, 2013-2016 


2013 2014 2015 2016 All 
(N=238) (N=268) (N=294) (N=105) (N=905) 
Increase ininterest = 32% 23% 24% 25% 26% 
Maintained high interest 43% 42% 52% 62% 48% 
_No change in interest 21% 31% 19% 138% = 22% 
Decrease in interest 4% 4% 4% 0% | 4% 


A11. Interest in STEM among adults, 2013-2015 


2013 2014 2015 All 

(N=23) (N=279) (N=17 1) (N=470) 
Increase in interest 9% 18% 15% 17% 
Maintained high interest 52% — 40% 55% 46% 
No change in interest 26% 37% | 27% | 33% 
Decrease in interest 13% 4% 2% 4% 


Confidence in science, technology, engineering, and math (STEM) 


This question is only included on youth surveys, so no adult data are available. 


A12. Confidence in STEM among youth, 2013-2016 


2013 2014 2015 2016 All 
(N=235) (N=187) (N=258) (N=103) (N=783) 
Increase in confidence 34% 33% 36% 26% | 33% 
Maintained high confidence 39% 38% 41% 54% 41% 
No change in confidence 23% 25% 21% 17% 22% 
Decrease in confidence 4% 4% 3% 3% 3% 
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Perceptions of stakeholders about 360 


This section includes more detailed descriptions of the measures included in the 
“operations” and “collaboration” measures tracked by 360. 


A12. Operations questions, Years 1-4 


2013 2014 2015 2016 
(N=25) (N=28-29) (N=20) (N=10) Graph 
Has adequate funding to complete the work. 4.00 4.00 3.75 3.70 _ ee 
Has the right partners involved in the work. 4.24 4.41 3.95 4.00 - A ee 
Has taken on the right amount of work. 4.00 4.07 3.70 3.50 _ 
Has the right governance structures to succeed. 4.32 4.17 4.05 4.10 ee 
Has the right leadership to succeed. 4.60 4.68 4.55 3.60 _ -\ 
Is progressing at the right pace. 3.92 4.14 4.10 3.80 aa 
Pea the right strategies to ensure he ee aa — Oe 
ee that would not otherwise be ie ee ie ae st 


Note. Mean scores based on responses to individual items; items were assigned the following values: “strongly agree’=5, “agree”=4, “neither agree 
nor disagree/neutral/no opinion”=3, “disagree”=2, “strongly disagree”=1. 


A13. Collaboration questions, Years 1-4 


2013 2014 2015 2016 

(N=25) (N=28-29) (N=20) (N=10) Graph 
hats of commitment among 360 partners is oa 135 ee ae a 

‘andtesponsbiltes 884383890 

peed indicat adiciak le dd 4.08 4.32 4.10 440 Zoe 7 
eee 360 will achieve its goals and i as rer or an 
amae dvereponisctvey, AAO Aah Aon 
Hse is effectively shared with those who jh rr he ee . i. 


Note. Mean scores based on responses to individual items; items were assigned the following values: “strongly agree”=5, “agree”=4, “neither agree 
nor disagree/neutral/no opinion”=3, “disagree”=2, “strongly disagree”=1. 
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Evaluation activities and methodology 


Over the course of the four-year grant period, the following evaluation activities (and 
corresponding methodologies) were completed: 


= Program logic model: Together with 360 staff, Wilder Research developed a logic 


model to capture the resources, activities, and projected outputs and outcomes of 360’s 
grant activities. 


Partner feedback: Wilder Research has conducted four partner surveys with 360 
partners. In the first year of the grant (February 2013) Wilder administered a web 
survey to 40 program partners identified by 360 staff; 30 people completed the 
survey, for a response rate of 75%. The first survey focused on partners’ expectations 
and understanding of the goals for the grant. In 2014, Wilder changed the 
methodology to get more in-depth feedback from partners, conducting 30 telephone 
interviews. The second survey asked partners to discuss whether their expectations 
had been met so far, the value the Center had created, their perspectives on the 
structure and partnerships of the Center, and any successes and challenges they have 
viewed to date. For the 2015 and 2016 surveys, Wilder returned to the web survey 
method. The 2015 survey was shared with 40 partners; 21 completed the survey, for a 
response rate of 53 percent. The 2016 survey was shared with 39 partners; 10 
completed the survey, for a response rate of 26 percent. While the questions related to 
collaboration and operations remained mostly the same across all surveys, much of 
the final survey focused on perspectives related to the work over the entire four years 
of the grant. 


Summer camp participant surveys: One important aspect of the 360 mission is to 
expose K-12 students to, and increase enthusiasm for, science, technology, engineering, 
and math (STEM) and manufacturing careers. Therefore, each year, 360 and its 
partners hold summer camps for youth. In order to evaluate the success of these camps 
in meeting their goals, 360 staff have distributed self-administrated questionnaires, 
designed by Wilder, to youth participants at the end of each camp. The completed 
surveys are then sent back to Wilder for analysis and reporting. The table below 
illustrates the number of respondents (and distinct camps that respondents attended) 
by year. 
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A17. Youth survey respondents and camps 


Number of youth Number of distinct camps 
survey respondents respondents attended 
Year 1 (2012) 59 3 
Year 2 (2013) 244 11 
Year 3 (2014) 184 9 
Year 4 (2015) 197 11 


= VEX Robotics youth and adult surveys: Like the summer camps, the VEX Robotics 
program is intended to build awareness and interest in STEM and manufacturing 
careers, as well as grow the pipeline of manufacturing students and workers in 
Minnesota. Teams of youth work together to build robots from kits and then compete 
with other teams at scrimmages and tournaments. Across most years of the grant, 
Wilder administered a web survey to youth participants of the VEX Robotics 
program, as well as the adults who were involved (i.e., parents, teachers, and 
coaches). The purpose of the survey was to gauge the success of the VEX Robotics 
program as well as the satisfaction of participating youth and adults. To reach youth, 
evaluators sent the youth survey link to the e-mail addresses of parents. Parents were 
asked to forward the link to their children, and both groups were instructed to forward 
the survey to other adults or youth who may have been involved. In years three and 
four of the grant, Wilder and 360 develop a consent form that adults could distribute 
prior to each tournament. Parents could either write their own email address or provide 
the email address of their child. 360 staff collected all completed consent forms and sent 
them to Wilder. Parents and coaches were also sent the youth survey link (when they 
received their own survey) to share with their students or children. This methodology, 
of using consent forms and sending the youth survey link to adults, is likely 
responsible for the increased number of respondents in the later years of the grant. 


A18. VEX survey respondents, youth and adults 


Number of youth Number of adult survey 
survey respondents respondents 
Year 1 (2012) n/a n/a 
_ Year 2 (April 2013) - 33 32 _ 
Year 2 (February 2014) 23 55 
_ Year 3 (February 2015) 74 76 _ 
Year 4 (February 2016) 112 161 
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= Youth outreach event tracking: In addition to the surveys administered to event 
participants, in year two, Wilder created and sent a “youth outreach event tracking 
form” via Survey Monkey to each school that hosted an event to track logistical 
information, such as how many youth participated, the age, gender, and race/ethnicity 
of participants, and other important information used in reporting to funders. While 
this was not in the original work plan, it has been useful to 360 over the past three 
years in tracking specific information about their youth outreach efforts. 


= Tour of Manufacturing participant and host surveys: Every fall, 360 and its 
partners coordinate manufacturing businesses across Minnesota to provide tours of 
their facilities for students, educators, job seekers, other manufacturers, and the general 
public. This event is called the Dream It. Do It. Minnesota Statewide Tour of 
Manufacturing. For the past four years, to help understand the impact of the tours, 
Wilder has administered a survey to both Tour participants and the businesses that 
hosted the tours. In the first two years of the grant, Wilder administered both surveys via 
SurveyMonkey, and Tour participants were directed to the online survey through a flyer 
distributed at each site. However, to address the large decrease in responses during year 
two, Wilder changed the methodology so that staff from each host business distributed 
hard copies of the survey (as a self-administered questionnaire) to participants. Hosts 
then returned completed surveys to Wilder Research in a pre-addressed envelope. 


A19. Tour of Manufacturing survey respondents, participants and hosts 


Number of participant Number of host 
respondents respondents 
Year 1 (2012) 117 21 
Year 2 (2013) 28 35 
Year 3 (2014) 391 34 
Year 4 (2015) 239 58 


= Tour of Manufacturing coordination survey: In years two through four of the grant, 
in addition to the host and participant surveys, Wilder also conducted a brief survey 
with local Tour of Manufacturing coordinators to help gauge the amount of resources 
that organizations contributed to their local Tour of Manufacturing effort. This was a 
web-based survey sent only to six to eight people, depending on the year. 


= Longitudinal “perceptions” questions: In year two of the grant, to consistently 
track perceptions of manufacturing and STEM over time, the evaluation team 
developed a series of questions to be used across all event surveys, such as the Tour 
of Manufacturing, summer camp, and VEX Robotics surveys. The questions were 
first used in the 2013 summer camp survey. 
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Analysis of administrative data from Integrated Statewide Record System 
(ISRS): Over the course of the four-year grant period, Wilder worked with 360 and 
Minnesota State, formerly MnSCU, to pull administrative data for the students and 
graduates of 360-related programs. 


Analysis of wage data from the Minnesota Department of Employment and 
Economic Development (DEED): During year three, Wilder received data on 360 
program graduate employment rates and wage information for the baseline years 
(2010-12) and first year of implementation (2013). These data will continue to be 
collected during the new grant period. 


Balanced Scorecard. In the third year of the grant, Wilder worked with 360 to design 
a Balanced Scorecard dashboard to align the program’s activities to its vision and 
strategy and to monitor performance. Staff selected the most important measures from 
an extensive list compiled over the first two years of the evaluation that fit in four 
distinct organizational perspectives — customer, financial, internal processes, and 
learning and growth. 


Teacher guide survey: In 2014, 360 sponsored the production and dissemination of a 
toolkit for teachers, called Your Future is Made in Manufacturing. To help understand 
how teachers used the toolkit and how the materials might be improved, Wilder sent a 
survey link to a list of 46 educators who had downloaded a toolkit online. In total, 22 
individuals accessed the survey. The survey was administered again in 2015, but with 
a very low response rate. A survey was emailed to 147 addresses that had requested 
the guide, now called An Introduction to Manufacturing in Minnesota Teacher Guide. 
Only 18 recipients (12%) responded to the first question of the survey. 


Employer survey: In the spring of 2016, just as the grant was ending, Wilder and 
360 sent a web-based survey to employers to hear about their experiences hiring 
graduates from manufacturing and engineering programs, and specifically about the 
qualities that are important to them in hiring recent graduates, and the preparedness of 
recent graduates who enter the workforce. A survey link was sent to 139 employers 
and 59 responded, for a response rate of 42 percent. 


Career Success Skills survey: 360 created 26 Career Success Skills learning 
modules to provide educators and industry with more resources to help students and 
employees develop their professional skills. To understand how people used the 
learning modules, Wilder sent a survey to Intermediaries — the faculty and 
professionals who access the learning modules to use with students and employees — 
and End Users — the students, employees, or others with whom the Intermediaries 
share the modules. A web survey was emailed to 75 End Users and 197 
Intermediaries; 10 End Users and 21 Intermediaries responded. 
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Graduation and majors data 


Wilder Research receives graduate data from Minnesota State’s ISRS data system. This 
analysis compares outcomes for 360’s ten original partner institutions—teferred to in this 
report as “360 Core” schools—to graduates in the same 360-affiliated programs (roughly, 
manufacturing-related programs) at other Minnesota State schools. Those ten original 
partner schools are identified in Figure A6. Of the original ten partner institutions, 360 
reports the number of majors for nine, all but Bemidji State. 


Gradually, 360 welcomed five new partners to the collaboration: Anoka-Ramsey Community 
College, Century College, Hennepin Technical College, Ridgewater College, and South 
Central College. For the purpose of this report, these new partners are grouped with 
“other Minnesota State schools,” as 360 has had less time to impact these partners. 


A20. College partners included in administrative data 


“360 Core” schools Majors data 
Anoka-Ramsey Community College No No 
Bemidji State University Yes No 
Central Lakes College Yes Yes 
Century College No No 
Hennepin Technical College No No 
Lake Superior College Yes Yes 
Minneapolis Community and Technical College Yes Yes 
Northland Community and Technical College Yes Yes 
Northwest Technical College (Bemidji) Yes Yes 
Pine Technical College Yes Yes 
Ridgewater College No No 
Riverland Community College Yes Yes 
Saint Cloud Technical and Community College Yes Yes 
Saint Paul College Yes Yes 
South Central College No No 
360: Cumulative Findings 35 Wilder Research, August 2017 


from a 4-year Evaluation 


